Skip to content

Refactor CatFile batch implementation and introduce batch-command for git 2.36 #34651

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 32 commits into
base: main
Choose a base branch
from
Draft

Refactor CatFile batch implementation and introduce batch-command for git 2.36 #34651

wants to merge 32 commits into from
+542 −347

Conversation

@lunny
Copy link
Member

@lunny lunny commented Jun 8, 2025 •
edited
Loading

For git version 2.36, git cat-file --batch-command was introduced which can replace git cat-file --batch and git cat-file --batch-check.

This PR implements an abstract layer for the batch commands so that both git 2.36 and lower version git can work.
If git version is lower than 2.36, it will start two subprocesses git cat-file --batch and git cat-file --batch-check.
If git version is greater than 2.36, only git cat-file --batch-command will be started.

This reduced half of child processes of git catfiles.

@lunny lunny added the type/refactoring Existing code has been cleaned up. There should be no new functionality. label Jun 8, 2025
@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Jun 8, 2025
@github-actions github-actions bot added the modifies/go Pull requests that update Go code label Jun 8, 2025
@lunny lunny added this to the 1.25.0 milestone Jun 8, 2025
@lunny lunny mentioned this pull request Jun 8, 2025
Closed
@lunny lunny closed this Jun 12, 2025
@GiteaBot GiteaBot removed this from the 1.25.0 milestone Jun 12, 2025
@lunny lunny reopened this Jun 12, 2025
@lunny lunny force-pushed the lunny/catfile_batch_refactor branch from f0b6480 to 334eb9e Compare June 18, 2025 17:15
@lunny lunny added this to the 1.25.0 milestone Jun 18, 2025
@wxiaoguang wxiaoguang removed this from the 1.25.0 milestone Aug 29, 2025
@lunny lunny added this to the 1.26.0 milestone Sep 29, 2025
@GiteaBot GiteaBot removed the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Oct 20, 2025
wxiaoguang
wxiaoguang previously approved these changes Oct 27, 2025
View reviewed changes
@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Oct 27, 2025
@wxiaoguang wxiaoguang enabled auto-merge (squash) October 27, 2025 00:20
@wxiaoguang wxiaoguang disabled auto-merge October 27, 2025 00:58
@wxiaoguang
Copy link
Contributor

wxiaoguang commented Oct 27, 2025 •
edited
Loading

Saw many flaky tests:


--- FAIL: TestPullCreate (0.71s)
    testlogger.go:61: 2025/10/27 00:39:18 HTTPRequest [I] router: completed GET /user/login for test-mock:12345, 200 OK in 7.8ms @ auth/auth.go:184(auth.SignIn)
    testlogger.go:61: 2025/10/27 00:39:18 HTTPRequest [I] router: completed POST /user/login for test-mock:12345, 303 See Other in 9.8ms @ auth/auth.go:197(auth.SignInPost)
    testlogger.go:61: 2025/10/27 00:39:18 HTTPRequest [I] router: completed GET /user1/repo1 for test-mock:12345, 404 Not Found in 3.4ms @ context/repo.go:417(context.RepoAssignment)
    testlogger.go:61: 2025/10/27 00:39:18 HTTPRequest [I] router: completed GET /user2/repo1 for test-mock:12345, 200 OK in 86.0ms @ repo/view_home.go:367(repo.Home)
    testlogger.go:61: 2025/10/27 00:39:18 HTTPRequest [I] router: completed GET /user2/repo1/fork for test-mock:12345, 200 OK in 27.3ms @ repo/fork.go:126(repo.Fork)
    testlogger.go:61: 2025/10/27 00:39:18 HTTPRequest [I] router: completed POST /user2/repo1/fork for test-mock:12345, 500 Internal Server Error in 22.5ms @ repo/fork.go:137(repo.ForkPost)
    repo_fork_test.go:54: Response length:  2439
    repo_fork_test.go:54: 
        	Error Trace:	/home/runner/work/gitea/gitea/tests/integration/integration_test.go:354
        	            				/home/runner/work/gitea/gitea/tests/integration/integration_test.go:160
        	            				/home/runner/work/gitea/gitea/tests/integration/repo_fork_test.go:54
        	            				/home/runner/work/gitea/gitea/tests/integration/pull_create_test.go:138
        	            				/home/runner/work/gitea/gitea/tests/integration/git_helper_for_declarative_test.go:88
        	            				/home/runner/work/gitea/gitea/tests/integration/pull_create_test.go:136
        	Error:      	Not equal: 
        	            	expected: 200
        	            	actual  : 500
        	Test:       	TestPullCreate
        	Messages:   	Request: POST /user2/repo1/fork
    testlogger.go:134: Flushing queues failed with error context canceled
=== TestPullCreate_TitleEscape (tests/integration/pull_create_test.go:164)
--- FAIL: TestPullCreate_TitleEscape (0.64s)
    git_helper_for_declarative_test.go:62: 
        	Error Trace:	/home/runner/work/gitea/gitea/tests/test_utils.go:236
        	            				/home/runner/work/gitea/gitea/tests/integration/git_helper_for_declarative_test.go:62
        	            				/home/runner/work/gitea/gitea/tests/integration/pull_create_test.go:164
        	Error:      	Received unexpected error:
        	            	failed to generate sequence update: context canceled
        	Test:       	TestPullCreate_TitleEscape
    testlogger.go:61: 2025/10/27 00:39:18 HTTPRequest [I] router: completed GET /user/login for test-mock:12345, 200 OK in 8.0ms @ auth/auth.go:184(auth.SignIn)
    testlogger.go:61: 2025/10/27 00:39:19 HTTPRequest [I] router: completed POST /user/login for test-mock:12345, 303 See Other in 10.1ms @ auth/auth.go:197(auth.SignInPost)
    testlogger.go:61: 2025/10/27 00:39:19 HTTPRequest [I] router: completed GET /user1/repo1 for test-mock:12345, 404 Not Found in 4.0ms @ context/repo.go:417(context.RepoAssignment)
    testlogger.go:61: 2025/10/27 00:39:19 HTTPRequest [I] router: completed GET /user2/repo1 for test-mock:12345, 200 OK in 86.5ms @ repo/view_home.go:367(repo.Home)
    testlogger.go:61: 2025/10/27 00:39:19 HTTPRequest [I] router: completed GET /user2/repo1/fork for test-mock:12345, 200 OK in 31.8ms @ repo/fork.go:126(repo.Fork)
    testlogger.go:61: 2025/10/27 00:39:19 HTTPRequest [I] router: completed POST /user2/repo1/fork for test-mock:12345, 500 Internal Server Error in 27.8ms @ repo/fork.go:137(repo.ForkPost)
    repo_fork_test.go:54: Response length:  2439
    repo_fork_test.go:54: 
        	Error Trace:	/home/runner/work/gitea/gitea/tests/integration/integration_test.go:354
        	            				/home/runner/work/gitea/gitea/tests/integration/integration_test.go:160
        	            				/home/runner/work/gitea/gitea/tests/integration/repo_fork_test.go:54
        	            				/home/runner/work/gitea/gitea/tests/integration/pull_create_test.go:166
        	            				/home/runner/work/gitea/gitea/tests/integration/git_helper_for_declarative_test.go:88
        	            				/home/runner/work/gitea/gitea/tests/integration/pull_create_test.go:164
        	Error:      	Not equal: 
        	            	expected: 200
        	            	actual  : 500
        	Test:       	TestPullCreate_TitleEscape
        	Messages:   	Request: POST /user2/repo1/fork
    testlogger.go:134: Flushing queues failed with error context canceled
=== TestPullBranchDelete (tests/integration/pull_create_test.go:226)

=== TestAPIChangeFiles (tests/integration/api_repo_files_change_test.go:63)
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:142 +0x447
goroutine 692 [select]:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc002960620)
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/util/buffer_pool.go:206 +0xc8
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool in goroutine 691
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/util/buffer_pool.go:237 +0x17a
goroutine 1179514 [sync.WaitGroup.Wait, 10 minutes]:
sync.runtime_SemacquireWaitGroup(0xc0047158c0?, 0x0?)
	/opt/hostedtoolcache/go/1.25.3/x64/src/runtime/sema.go:114 +0x2e
sync.(*WaitGroup).Wait(0xc001f30320)
	/opt/hostedtoolcache/go/1.25.3/x64/src/sync/waitgroup.go:206 +0x85
code.gitea.io/gitea/tests/integration.TestRepoCommitsStatusParallel(0xc0085fddc0)
	/home/runner/work/gitea/gitea/tests/integration/repo_commits_test.go:208 +0x2aa
testing.tRunner(0xc0085fddc0, 0x40e0668)
	/opt/hostedtoolcache/go/1.25.3/x64/src/testing/testing.go:1934 +0xea
created by testing.(*T).Run in goroutine 1
	/opt/hostedtoolcache/go/1.25.3/x64/src/testing/testing.go:1997 +0x465
goroutine 653866 [select, 16 minutes]:
code.gitea.io/gitea/modules/log.(*EventWriterBaseImpl).Run(0xc00806b860, {0x457a4f8, 0xc0040b94c0})
	/home/runner/work/gitea/gitea/modules/log/event_writer_base.go:81 +0x279
code.gitea.io/gitea/modules/log.eventWriterStartGo.func1()
	/home/runner/work/gitea/gitea/modules/log/event_writer_base.go:157 +0xa5
created by code.gitea.io/gitea/modules/log.eventWriterStartGo in goroutine 653864
	/home/runner/work/gitea/gitea/modules/log/event_writer_base.go:153 +0x191

@wxiaoguang wxiaoguang marked this pull request as draft October 27, 2025 01:26
@wxiaoguang wxiaoguang force-pushed the lunny/catfile_batch_refactor branch from 81fefd8 to f09c8a1 Compare October 27, 2025 02:17
@wxiaoguang
Copy link
Contributor

wxiaoguang commented Oct 27, 2025 •
edited
Loading

It seems that the similar failures also occur in old changes (before my refactoring), for example:

image

The code before this PR uses two separate processes, so the "readers" won't conflict.

But this PR's approach uses only one process, "contents" and "info" share the same reader, it is very fragile, and if it is abused, then deadlock. Just a guess.

@wxiaoguang wxiaoguang force-pushed the lunny/catfile_batch_refactor branch from f09c8a1 to 3e5b6c3 Compare October 27, 2025 03:25
@lunny
Copy link
Member Author

lunny commented Oct 27, 2025

Maybe we can merge the two methods, Write and GetReader, into a single function and introduce an internal lock mechanism to prevent potential concurrency issues.

@wxiaoguang wxiaoguang force-pushed the lunny/catfile_batch_refactor branch 2 times, most recently from aea9e48 to 039fa34 Compare October 27, 2025 04:14
@wxiaoguang wxiaoguang force-pushed the lunny/catfile_batch_refactor branch from 039fa34 to d8997a3 Compare October 27, 2025 04:43
@wxiaoguang wxiaoguang force-pushed the lunny/catfile_batch_refactor branch 2 times, most recently from 1c54fb5 to c554e85 Compare October 27, 2025 05:36
@wxiaoguang wxiaoguang force-pushed the lunny/catfile_batch_refactor branch from c554e85 to f4ec9d9 Compare October 27, 2025 06:07
@GiteaBot GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. labels Oct 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wxiaoguang wxiaoguang wxiaoguang left review comments

+1 more reviewer

@Zettat123 Zettat123 Zettat123 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

lgtm/need 1 This PR needs approval from one additional maintainer to be merged. modifies/go Pull requests that update Go code modifies/internal performance/cpu type/refactoring Existing code has been cleaned up. There should be no new functionality.

Projects

None yet

Milestone

1.26.0

Development

Successfully merging this pull request may close these issues.

4 participants

@lunny @wxiaoguang @Zettat123 @GiteaBot